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COMPARISON AND IDENTIFICATION 



Preview 



Purpose ; 

To compute student engaged times for each classroom for reading/ language 
arts and for math, to compare Information collected In classrooms with 
research findings, to identify opportunities for Improvement, and to 
set tentative goals for student engaged time, allocated time, and en- 
gagement rate* 

Objectives : 

To teach participants to find their own average student engaged times 
for reading/ language arts and for math. 

To provide a rationale for and overview of the activities associated 
with the Comparison and Identification Phase of the instructional 
improvement cycle. 

To teach participants to select appropriate graph(s) or data base for 
comparison. 

To teach participants to read and interpret graphs, deciding whether 
to change student engaged time and. If so, setting a tentative goal. 

To teach participants to examine allocated time and engagement rate in 
terms of their own situations, setting tentative goals for these vari- 
ables and revising the tentative goal for student engaged time if 
necessary. 

Expected Outcomes : 

Average student engaged times for reading/ language arts and for math 
for each classroom; identification of opportunities for improvement; 
goals for student engaged time, allocated time, and engagement rate 
for both reading/ language arts and mathematics for all teachers. 

Time ; 

About 2-3 hours o£ meeting time is needed for an in-depth presentation 
of all topics. This time is allocated by topics as follows: 

A. Agenda: Rationale, activities, and outcomes (10 minutes) 

B. Summary Sheet for Time (45 minutes) 
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C, Selecting a datn hase (10 minutes) 

F). Comparing data to rcBcarch (30"50 mlnutOH) 

E, Making a decision (30-50 minutes) 

The time needed to complete the starred minimal activities Is about 
2^3 hours , 

Materials ! 

Completed Allocated Time Log(s) and Engagement Rate Fonn(s) for each 
classroom 

Agenda 

Major Activities Chart 

Handouts: 2H4. 2H6. 2H7, 2H9, 2H12, 2H14, 2H16, 2H22, 2H23, 2H24, 2H26 

Transparencies: 2T1-2T3. 2T5, 2TB, 2T10-2T13, 2T15» 2T19-2T21, 2T25 

Checkpoints (for leaders only); /2H17 

Answer Keys for Checkpoints: 2T18 

Calculators 

Rulers 

Overhead projector and screen 
Follow-up : 

Make arrangements for the Selection and Preparation meeting. Find 
out about national, regional, and local resources for strategies. 
Leaders may wish to develop additional handouts describing these 
resources for participants. Remind teachers to being schedules and 
completed Engagement Rate Forms with them to the next meeting. 

Note; Leaders may wish to provide multiple copies (or a reusable 
laminated copy) of the nomograph (2H4). 
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COMPARISON AND IDENTIFICATION 
AGENDA 

A. REVIEW OF AGENDA 

• Rationale, activities* and outcomes 

B. SUMMARY SHEET FOR TIME 

•Explanation of how to determine student engaged time 

•Calculation of student engaged time in reading/language 
arts and math for each classroom 

C. SELECTING A DATA BASE 

•Background considerations 
•Some selection guidelines 

D. COMPARING DATA TD RESEARCH 

•Parts of the graphs 
•Reading the graphs 

•Setting a tentative goal for student engaged time 

E. MAKING A DECISION 

• Setting feasible tentative goals for engagement rate 
•Computing tentative goals for allocated time 
•Deciding if allocated time goals are feasible 
•Recording and/or revising goals 
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MAJOR ACTIVITIES FOR COMPARISON AND IDENTIP'ICATION 



OUTCOMES! 



OBJECTIVES OF 
MAJOR ACTIVITIES: 



Parclclpancs eecabllsh goals for scudenc engaged 
time, engagement rate, and allocated time from 
their own data. 



Participants set a tentative goal for allocated 
time, examine that goal In light of their own 
circumstances, and then make any necessary 
adjustments, given their tentative goals for 
student engaged time and engagement rate. 



Participants set a tentative goal for engagement 
rate, given Information on engagement rate 
research • 



Participants compare their classroom Information 
to the research and set a tentative goal for 
student engaged time* given Information about 
the graph • 



Participants select the data base pertinent to 
their classrooms, given Information about data 
bases. 



Participants calculate student engaged time 
and complete the Summary Sheet for Time from 
completed Allocated Time Logs and the 
corresponding Engagement Rate Forms, 
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CloBBarjLfor Comparison and Identification 



Above Expected Level of 
Achievement 



At Expected Level of 
Achievement 



Below Expected Level of 
Achievement 



Data Base 




Difference from Expected 
Raw Score 



Inverted U Graph 



Minimum Change Unit 



Positive Linear Graph 



Raw Score 

Summary Sheet for Time 
Zones 
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Difference from expected raw score 1b posi- 
tive* Students have higher posttest scores 
than expected* 

Difference from expected raw score Is zero* 
Students have posttest scores equal to what 
was expected* 

Difference from expected raw score la nega- 
tive. Student8 have lower posttest scores 
than expected* 

Research study providing specific Informa- 
tion about a variable (e*g*, for student 
engaged time the data bases are Stalllngs £• 
Kaskowltz and the Beginning Teacher Evalua^ 
tlon Study)* 

Difference between actual posttest score and 
posttest score as predicted from pretest* 
This difference Is used as a measure of 
student achievement* 

Graph shaped like an Inverted U (n), Imply- 
ing that a modv rate amount of a variable 
(such as student engaged time) corresponds 
to the highest achievement levels* 

Smallest change In student engaged time to 
be made In setting a tentative goal for 
student engaged time* 

Straight line graph Indicating that Increases 
In a variable (such as student engaged time) 
are accompanied by Increases In student 
achievement (Ite*, more Is better)* 

Number of correct answers* 

Form used to summarize collected Information 
on allocated time, engagement rate, .ind 
student engaged time* 

Ranges on a scale (such as student engaged 
time) that correspond to ranges on another 
scale (such as student achievement)* Zones 
are determined according to the relation- 
ship (shape of the graph) between these two 
variables* For each graph Included In this 
section, positive, zero, and negative zones 
are specified on the student engaged time 
scale, which correspond to positive, zero, 
and negative differences from expected raw 
scores on the student achievement scale. 
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Materlalfl for Conparlflon and Identification 



The following materials are Included In thla section: 



Topic A 

*2T1 
★ 2T2 

2T3 

Topic B 

2H4 

*2T5 
(a-b) 
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★ 2H6 

(a-b) 

(l) 2H7 
(a-h) 

© 2T8 
^ (a-b) 

Topic C 

★ 2T10 

Topic D 

★ 2T11 

★ 2T12 

★ 2T13 
*2H14 
iA^2T15 

i 

*2H16 

©/2H17 
(a-b) 

@ 2T18 



Now What Do X Do With Hy Data? 
Comparison Phase Activities 
Student Engaged Time Formula 



Time Nomograph 

Completed Summsry Sheet for Time 
Summary Sheet for Time 
Summary Sheet for Time 
Answer Key for 2H7 



Some Selection Guidelines 



First Grade Reading and Word Analysis Total Graph 
Third Grade Reading and Language Total Graph 
Reading a Graph 
Reading the Graphs 

Considerations In Setting a Tentative Goal 
Comparing Your Data to Research 
Comparing Data to Research 

Key for /2H17 



— Minimal activity for all participants 
(l) — Minimal activity for leaders only 
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Topic E 

■A'2T19 Making a DcclBlon 

2T20 Different Waya of Getting 90 Mlnutca of Student Engaged Time 
-A'2T21 Setting Engagement Rate Goala 

2H22 Tony Caalnl'a Goala 

J(a-c) 
★2H23 Making a Declalon 

© 2H2A Making a Declalon 
(a-c) 

Q 2T25 Key for 2H2A 
(a-b) 

Graphs 

*2H26 Graphs 
(a-f) 





-k — Minimal activity for all participants 
^ — Minimal activity for leaders only 
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Notes for Comparloon and Identification 

The following notes are Included In this section; 

D.l.d. Student Achievement — Difference from Expected Raw Score 

D.l.e. Minimum Chungc Units 

D. 1. f . Zones 

C.2.b. Guidelines for Setting an Engagement Rate Goal 
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A. Agenda: Rationale, acclvlcles, and outcomes (10 minutes) 



Rationale. Participants now have data on their own classrooms and 
are anxious to answer the question, *'How am I doing?" The rationale 
of, activities In, and outcomes of a session addressed to that 
question are outlined. The overview and discussion of the agenda 
provide a way of organizing the meeting for all present. 



Materials 

*2Tl~Now What Do I Do 
With My Data? 



'^2T2 — Comparison Phase 
Activities 
— Agenda 
2T3 — Student Engaged 
Time Formula 



Strategy 

Briefly review events In Information 
Collection . Provide rationale for 
comparison phase. 

Outline major activities In this 
phase. Review major activities — 
computing student engaged time, 
selecting a graph, reading a graph or 
data base, setting goals. Address 
questions/concerns about the activi- 
ties planned for the session. Review 
by stating the expected outcomes. 
Then show how those outcomes lead to 
the final two phases of the Improve- 
ment cycle. 



★ 1. Rationale <2T1) 
a. 



(1 minute) 



Participants have data—^amount of allocated time and engagement 
rate In his/her class 

b. Participant's question — So what? What kind of achievement can 
be expected given my values for allocated time and engagement 
rate? 

c. Answer to questions—compare student engaged time values with 
research data 

d. Decision for action based on: 

(1) Unique classroom circumstances/ values 

<a) Constraints (e.g., allocated time controlled by 
bell, schedule, clock, changing classes , etc.) 



(2) 



(b) Values — how much time can be allocated 
Implications of research 
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^2* Purpose of session (J minute) 

A* Compare data collected In classroom with data from research 

b. Set goals for student engaged time desired In reading/ 
language arts and mathematics 

'A' 3. Proposed agenda (2T2» Agenda) (3 minutes) 

a* Calculating student engaged time 

b* Selecting a data base 

(1) Data differ from grade to grade and subject to subject 

(2) Therefore, guidelines provided for selecting graph or 
data base 

c* Reading a graph 

(1) Terms to understand 

(2) Practice In reading and Interpreting 
d. Making decisions about student engaged time 

(1) Research and teacher Judgment suggest a tentative goal 

(2) Examine engagement rate and allocated time to determine: 

(a) How a change In student engaged time might be 
accomplished (2T3) 

<1) Change engagement rate 

(11) Change allocated time 

(111) Change both allocated time and engagement rate 

(b) Whether or not a change In allocated time or engage- 
ment rate Is desirable 

(3) Revised goal for student engaged time, allocated time, and 
engagement rate — based on research, teacher's knowledge/ 
values/judgment, constraints 

-^4* Discussion of agenda (3 minutes) 

5.^ Expected outcomes — participants will have; (1 minute) 

a. Calculated student engaged times for reading/ language arts and 
for math 

b. Decided whether or not they wish to change the student engaged 
time, allocated time, or engagement rate 

c. Established goals for student engaged time In reading/language 
arts and mathematics 
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ifrfi. Next steps (1 minute) 



a. Select modification or strategy for attaining goals 

(1) Review data collected 

(2) Review strategies from research, other teachers, 
brainstorming 

(3) Describe modification and plan steps for implementing 

b. When laodiflcation is in place« collect data on student 
engaged time 

(1) Data tell if and to what extent goals are met 

(2) Can lead to changes in strategy 



\ 




2.11 



15 



2.12 



7/28/80 



NOW WHAT DO 1 



DO WITH MY DATA? 



So what ? 




MARY SHEET FOR 009 
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Engagement 


engaged 
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WiPARISON PHASE ACTIVITIES 



FINAL 
OUTCOME 



REVIEW THE STEPS IN THE COMPARISON PHASE. 



I 



COMPUTE STUDENT ENGAGED TIME. 



I 



SELECT THE DATA BASE RELEVANT TO INDIVIDUAL 
CLASSROOM CHARACTERISTICS AND INTERESTS. 



I 



LEARN TO READ AND INTERPRET GRAPHS AND SET 
A TENTATIVE GOAL FOR STUDENT ENGAGED TIME 
BASED ON TEACHER JUDGMENT AND RESEARCH. 



1 



SET A TENTATIVE GOAL FOR ENGAGEMENT RATE. 



I 



SET A TENTATIVE ALLOCATED TIME GOAL. 
EXAMINE THE GOAL IN THE CONTEXT OF A GIVEN 
SCHOOL SETTING. 



I 



MAKE ADJUSTICNTS IN TENTATIVE GOALS FOR 
ENGAGEMENT RATE AND ALLOCATED TIME SO THAT 
FINAL GOALS FOR ENGAGEMENT RATE, ALLOCATED 
TIME, AND STUDENT ENGAGED TIME ARE 
ESTABLISHED. 
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STUDENT ENGAGED TIME FORMULA 



Student Engaged Time » Allocated Time X Engagement Rate 
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3. Summary Sheec for Time <45 mlnuces) 



Rationale . Parclclpancs need to learn how to find student engaged 
time by using the time nomograph or calculator and the collected 
data on allocated time and engagement rate. Teachers need to com- 
pute their own student engaged time to compare to research data. 
Leaders need to be sure participants understand how to compute 
student engageii time. 



Materials 

—Calculator 
2H4 — Time Nomograph 

^ 2T5 —Completed Summary 
<a-b) Sheet for Time 



^ 2H6 — Summary Sheet for 
(a-b) Time 

—Completed Allocated 

Time Logs 
— Completed Engagement 
Rate Forms 

(l) 2H7 — Summary Sheet for 

<a*h) Time 
(l) 2T8 —Answer Key for 

<a"b) 2H7 



Strategy 



Discuss how to use the nomograph. 

Describe the Summary Sheet for Tlme# 
stressing that It can he used for 
sex^eral sets of observations (see 
note B. 2) . 

Have teachers complete their own 
Summary Sheets. 
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Help leaders to complete 2H7, using the 
nomograph or calculator to find student 
engaged time. Give directions for each 
step as needed. Check and discuss answers. 
If an answer is incorrect, try to give a 
hint that will help the person correct the 
error (e.g., you skipped a column, how 
about the top, look at this column again). 
If procedures used were incorrect or 
numerous arithmetic errors were made, 
correct these errors. 



Review the use of the Summary Sheet 
in unusual circumstances with leaders. 



(L) 2H8 — Using the Summary 
Sheet in Depart- 
mentali:sed 
Situations 

Alternative Strategies 

1. Participants who need additional practice in using the nomograph 
<2H4) should complete worksheets CH14 and/or CH15 in Appendix C 
of Section I. 

2, Have participants needing practice In completing the Summary 
Sheet use numbers from previously completed activities in 
Section I, such as IH38 and/or IH40 (Allocated Time Log) and 
IV32 and IV33 (Engagement Rate Form). 



2.16 



20 



Uslnf^ the nomograph f 2HA) (5 mlnutos) 

a. Purpose—tool for multiplying alloc/itecl time by engagemoiU 
rate to get student engaged time 

(1) Can use to estimate answer 

(2) Can use calculator Instead 

(3) Can use to estimate/check work on calculator 

b. Procedures 

(1) Need straightedge— ruler or sheet of paper 

(2) Locate allocated time on allocated time scale 
(120 minutes) 

(3) Locate engagement rate on engagement rate scale (75%) 

(4) Place straightedge on point for allocated time and for 
engagement rate 

(5) Point where straightedge crosses student engaged time 
scale Is student engaged time (about 90 minutes) 

Summary Sheet for Time (10 minutes) 

a. Purposes-record classroom Information on time throughout the 
year 

b. Format 

(1) One sheet used to record information on time for reading/ 
language arts and for math for the whole year 

(a) If you teach the same subject to more than one class, 
you may wish to use different sheets for each class 

(2) Information at top 

(a) State, district, school^ and teacher — use codes if 
desired 

(b) Year 

(3) One line used for each day of observation f 2T5a ^ 
(a) One observation made on first day 

(1) Date observed (10-5) 

(11) Coder number (ED28) ^ part of period (beginning), 
and engagement rate (75%) from Engagement Rate 
Form 

(ili) Allocated time from Allocated Time Log 

•Use total for all activities (134 minutes) 

•If students receive instruction in a subject 
area from another teacher in addition to 
yourself (e.g., you teach reading and Tony 
Casini teaches language arts), add the 
estimated allocated time for that teacher* s 
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classroom to your own and record 
(see 2H9) or observe that class, 
student engaged time, and add It 



the total 

compute* 

to your own 



(Iv) 



Student engaged time (101 minutes) 



♦Use nomograph or calculator 



♦Allocated time x engagement rate 
minutes x . 75) 



(134 



(b) Two 
(1) 
(11) 



observations made on second day 
Date (10-9) 



Allocated time 



♦Record total allocated time from all 
activities on front (171 minutes) 

♦Record brief description of each activity 
and time In minutes for each In small 
table on back (TtSb) 



(111) Coder numbers (ED28 and XX36) , parts of period 
(beginning and middle), and engagement rates 
(70% and 68%) from Engagement Rate Form on 
front 

(Iv) Engagement rate 

♦Use table on back to record engagement 
rates for each activity observed — 70% for 
reading seatwork, 68% for spelling 

♦If an activity was not observed, use either 
average or engagement rate of most similar 
activity— used average of 69% for reading 
groups (circled) 

(v) Student engaged time 

♦Calculate for each activity (time In minutes 
X engagement rate) 

•Add to get total and record on front (119 
minutes) 

(c) Two observations made on third day 

(1) Same procedures as on previous day except ^ol 
missing engagement rates 

(11) For missing engagement rates In small table on 
back« used engagement rate of most similar 
activity rather than average (2T5b) 



♦Used reading groups engagement rate for 
reading seatwork (70%) 



♦Used spelling engagement rate for language 
(80%) 
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Averngc student cnfsaged tlmc-add all cntrtow 
in hI^ikU'iiI ('ngri^oil time r<^lumn nnd dlvhk' by 

[ 3 '**^) 

3. Own data (2H6) (20 minutes) 

a. Fill In Information at top on 2H6a 

lAr 3 or 4 b* Record date(6) observed^ coder number(s), part(fl) of pcirlod, 
and engagement rate(6) from Engagement Rate Forms 

c. Record allocated tlme(8) from Allocated Time Log(s) 

d* Find student engaged tlme(s)-"'allocated time x engagement 
rate 

e* Find average student engaged time for each subject aren 

(D^- Example (2il6, ML. ilS) (20 minutes) 
lAr 3 or 4 a* Fill In Information at top on 2H6a 
b* Reading/ language arts 

(1) October 8— only one observation made 

(a) Record coder number, part of period, engagement 
rate, and allocated time 

(b) Find studert engaged time — allocated time x 
engagement rate 

(2) October 9— two observations made 

(a) Record date, coder numbers, parts of oerlod, 
engagement rates^ and total allocated time on 
front of 2H6 

(b) Record each activity, time In minutes, and 
engagement rate In small table on back 

(c) Find student engaged time for each activity, 
add to get total, and record on front 

(3) October 10 — two observations made so same procedure 
as for October 9 

(4) Average student engaged time — add student engaged time 
column and divide by number of entries 

c. Math 

(1) First two days — only one observation made so same as 
procedure for reading/ language arts on October 8 

(2) Last day — get Information from completed forms 

(l)5* Use of Summary Sheet In departmentalized situations ( 2H9 ) (5 minutes) 

a. Situation A — groups of students are same for reading and language 
arts 

b* Situation B — groups of students vary for reading and language 
arts 

2.19 
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TIME NOMOGRAPH 
Allocated Time 
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State A+\c^nVi 



District Eas+ern 



School Neu>'t>e\p€n 
Teache r Cas'vf\'\ 



State # 



District # 



SUWARY SHEET FOR TIME 

School § 

Teacher § 



Year I^^I^JHO 



READING/LANGUAGE ARTS 



Date 


Coder # 


Part of 
Period 


Enqaqeinent 
Rate 


Allocated 
Time 


Student 
Engaged 
Time 


Average 
Student 

cn^gcQ 1 line 
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Date Subject Regii'mA / LAnavjoae. 





Activity 


Txme xn 
Minutes 


Engagement 
Rate 


otudent 
Engaged Time 


1 










2 
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Date fO't(> Subject Reo-A'^n^ / Uxn^v>a,q^t. 
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Time in 
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Student 
Engaged Time 
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State 

Distrlct_ 
School 



Teacher 



State //_ 
District 



SUMMARY SHEET FOR Tiy.E 



School // 



Teacher // 



Year 
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READING/UNGUAGE ARTS 



Date 


Coder // 


Part of 
Period 


Engagement 
Rate 

— 


Allocated 
Time 


Engaged 


Ave raj^e 
Student 
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MATH 



Date 


Coder # 


Part of 
Period 


Engagement 
Rate 


Allocated 
Time 


Student 
Engaged 
Time 


Average 
Student 
Engaged Time 
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Student Engaged Time " Allc»cated Tine x EDgagemenc Rnce 



DIRECTIONS 

1< Us« chl9 fonn to sunmiarlj&e data chroughouc che school year* 
2m Fill In the Information at che Cop of the form. 

3. Record che dece(s} of observation. One line on che form Is used for each day 

4. Record che cotal aljocaced clme (In mlnuces) from the Allocated Time Log for 
each subjecc. 

5. Record the coder numbera, pares of che period observed^ and engagement races 
for all observaclona made on thac dace. 

If Oftly one observation wea made in a subject area (readlng/laiiguage 
arta and/or math)t 

'Record the engagement rate from the Engagement Rate Form. 

• Find che scudent engaged time by using the nomograph C2H4) 
or a calculator (allocaced cime X engagemenc race). 

If more Chan one observacion was made In a stibjecc area^ ccmplece one of che 
tables below: 

• Record Che cime In minutes for each accivlcy In che subjecc 
area from che lasc column of che Allocaced Time Log. 

•Record Cite engagemenc races for each accivlcy observed 

from che Engagemenc Race Forma. 

If an actlvicy ws not observed^ record elcher che 

average of che engagemenc racea from the obaerved 

accivitles or che engagemenc race of che mosc similar 

observed accivlty. 
•Find che scudenc engaged cime for each accivlty by using 

che nomograph C2H4} or a calculator (cime In mlnuces x 

engagemenc race), 
•Add the scudenc engaged cine column and record the cocal 

oD the fronc of chls form, 

6. Find che average acudent engaged cime for all observaclona by adding all of che 
entries in the student engaged time column and dividing by the nuTiiber of entries 
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Time in 
TflntiteJ 


Enftag^ent 
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SUMMARY SHEET FOR TIME 




Jerome Kenyatta, one of the fifth grade teachers at New 
Delpen Elementary School, has collected information on 
P allocated time and engagement rate in his classroom for 
three days, October 8-10. He believes that the data 
obtained on these days are representative of activities 
.-^nV\ /\\\^ in his classroom. 

^Complete the information at the top of the Sumary Sheet. 

State— 03 
District — 47 
School— 08 
Teacher~K8KE 
Year— 1979-1980 

READING/LANGUAGE ARTS 



First Day (10/8) 

^Onty one observation was made, so record the following information 
on the front of the Simnary Sheet. 

Coder — XX36 

Part of Period — Beginning 
Engagement Rate — 64% 
Allocated Time — 170 minutes 

^Find the student engaged tim (allocated time x engagement rate). 

Second Day (lO/9) 

•JVo observations were made on October 9, so record the following 
information on the front of the Simnary Sheet, 



Coder if 

XX36 
1176 



Part of Period 

Middle 
End 



Engagement Rate 

60% 
66% 



Allocated Time 



) 



175 minutes 



^Use one of the small tables on the back to record the time in minutes 
and engagement rate of each activity on October 9* 



Activity 

Reading groups 
and seatwork 
Spelling 



Time in Minutes 
143 minutes 

30 minutes 



Engagement Rate 
60% 

66% 
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• Find the student engaged time for each activity (time in minutes x 
engagement rate) arid add to find the total, 

• Record the total student engaged time on the front of the form. 
Third Day (10/10) 

^Ttoo observations were also made on October 20* Record the date, 
coder numbers, parts of period observed, engagement rates, and 
total allocated time on the front of the Stmnary Sheet* 



Activity 

Reading groups 
Language seatwork 
Sustained silent 
reading 



Time in 
Minutes 

120 minutes 
30 minutes 

17 minutes 
167 minutes 



Engagement Coder Part of 

Rate 9 Period 

no observation 

70% 1176 Beginning 

66% XX36 Middle 



• Use another small table on the back to record the time in minutes and 
engagement rate for each activity. Use the language seatt^rk engage- 
ment rate (70%) for the missing engagement rate (reading groups). 
Find the total student engaged time* 

• Record the total student engaged time on the front of the Sumary 
Sheet, 

Average 

•Find the average student engaged time for reading/language arts over 
the three days of observation. 



MATH 

First Two Days (10/8 and 10/9) 

The data for Jerome's classroom are shown below: 

Date Coder // Part of Period Engagement Rate Allocated Time 

10/8 XX36 Beginning 80% 50 minutes 

10/9 1176 Middle 78% 60 minutes 



•Record this information and find the student engaged time. 
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Third Day (10/10) 



• On the front of the Stmnary Sheets record the date and total allocated 
time from the Allocated Time Log (/2H7d). 

• On the front of the Stmnary Sheets record the coder numbers^ parte of 
period observed and engagement rates for both of the Engagement Rate 
Forms completed on October 10 (/2H7e-h). 

•Since tuo observations were made on this date^ complete a small table 
on the back of the Summary Sheet: 

— Describe each activity and indicate the time in minutes for each 

(from the Allocated Time Log) ^ 
— Record the engagement rate of each of the observed activities^ 

using the time of observation to place each correctly* 
— Use the average engagement rate for the observed activities for 

any missing erigagement rates* 
— Find the student engaged time for each activity (time in minutes x 

engagement rate)* 
-^^Find the total student engaged time* 

•Record the total student engaged time on the front of the Summary 
Sheet* 




Average 

^ Find the average student engaged time for math. 
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STATE 

DISTRICT 



STATE H 0 3> 



DISTRICT f^jTL 



ALLOCATED TIME LOG 

SCHOOL a 0% DATE lo/jo 
TEACHER # K%<€ GRADED 



SUBJECT 

KO, OF STUiOiTS 
PRESENT ^ 
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TIME 
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Key for 2H7 

SUMMARY SHEET FOR TIME 




District £oiaAerr\ 

Schoo l Mt^'bt.ipen 1 state ^ School i f QS 

Teacher Ken^^o^tVo. 



2T8a 
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Stodcnt Engiigf^il Iliac * Atloc^itect TXine x Engngooicnt Ratu 



DIR£CriOHS 



2T8b 

Key for 2H7 




1. Use this torn to suttiff^ntly.c duLa thcoo^jhoot the school year. 

2. Fill In the infotmxttiofi .it tU^ top of thu forut. 

3. Recor<l the dace(fi) of obsocv.it lt>n * One line on the form Is uHt'd for each dny* 
Recor<l the tot.il allocated cimii Ctn minutes) from tho Allocociid time L^g tot: 
each subject * 

5* RiicnrJ the co<lct numbors, pacts of the piitiod observed, and eng^ageotent r^ices 
for all observations* made on that date* 

If only one t>bserv*Tr Ion wjs oiacJe in a subject area Creadtn^/l^iugu.ige 
arts and/ur mattO: 

•Record the (nJ^nr.ement rntu Hvorn the Ensagemeixt Rate Torm* 

* Find the studLUit cne^ised time by aslng the nomograph (2H4) 
or o calcolatot Collocated time x engagement rate)* 

If iDore chan one observation was auide In a sobject area, complete of the 
tables be low i 

•Record che time in mLnotes for each activity In the subject 
area fci>m the Ijst column of the Allocated Time Log* 

*Recotd t\w enj;arement rates Cor each activity obsor\/ed 
from the Eng^^jeiiieiit Rate Forms* 

If an activity was not observedj record either the 
average of thti fla**iigernent t^itest fcom Che obfrerved 
activities or the ^^n^^aijeinent r;»te of the most similar 
Observed activity, 
*Pind tijc f*cud*^nt enf^^jged time for each activity by onjiif;; 
the nomograph C2K4> or a calcolator (time In minuter k 
entj;;tp,ement raCe), 

• Add the stoderit ^^tl^5a^:^^d time column and recocd the total 
on the front of this form* 

6. Find the averagt> student engaged time for all observations by adding all of the 
entries in the stodent eng.jp.iid time column aad dividing by tho no-aber of entries* 
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USING THE SUMMARY SHEET IN DEPARTMENTALIZED SITUATIONS 



Situation A 



Ann Riley, Peter Demetrios, and Maria Malenkoi teach a group of about 
85 fifth and sixth graders divided into three classes* Ann teaches 
reading and social studies to each class, Peter teaches language arts and 
health, and Maria teaches math and science* The three teachers jointly 
complete three Summary Sheets, one for each class* Ann records allocated 
times and engagement rates for reading, Peter records this Information 
for language arts, and Maria records data for math* Ann and Peter use 
the tables on the back of the Summary Sheet whenever the same group of 
students is observed in both classrooms* Otherwise, they add together 
their allocated times for reading/language arts and use the engagement 
rate for whichever class was observed* 



Situation B 




Susan Brown teaches five different groups of students; she has two 
average seventh-grade English classes, one seventh-grade remedial English 
class, and two advanced seventh-grade English classes* Students also 
attend reading classes for 30 minutes each day* Students are grouped 
differently for instruction in reading, however, so Susan cannot combine 
her data with anyone else^s. She decided earlier to collect Information 
for only two classes: the remedial class and one of the average classes* 
She uses two Summary Sheets, one for each class, to analyze her data* In 
finding allocated Hime, she adds the 30-minute reading period to her own 
time to record a total before multiplying to find the student engaged time* 
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T.3. Summary Sheet for Time 

The Summary Sheet for Time ts used to record classroom information 
on time throughout the year. In general » each teacher uses only one 
sheet. However, if a teacher has more than one group of students for 
the same subject (e.g., two or more different math classes), he/she 
may wish to use different sheets for each class. 

,The information at the top of the form may be recorded even before 
any information is collected. For each day on which observations were 
made, the teacher first records the date. In the example on 2T5, there 
was only one observation made in reading/ language arts on the first 
day, October 5. Next to the date, the teacher records the coder number 
of the observer (ED28), the part of the period observed (beginning) and 
the engagement rate (75%) from the Engagement Rate Form. Then the 
teacher records the total allocated time for all activities from the 
Allocated Time Log (134 minutes). The teacher next uses either the 
nomograph or a calculator to find student engaged time (allocated time 
X engagement rate = 134 minutes x .75 - 101 minutes). 

On the second day of information collection, two reading/language 
arts observations were made. The teacher first records the date (10-9), 
the total allocated time for all activities from the Allocated Time 
Log (171 minutes), and the coder numbers (ED28 and XX36), parts of the 
period observed (beginning and middle), and engagement rates (70% and 
68%) from the Engagement Rate Form. Then, since two observations were 
made on the same day, he/she uses one of the small tables on the back 
of the Suraraary Sheet to record a brief description of each activity 
and its time in minutes from the Allocated Time Log and the engagement 
rates for all activities observed from the Engagement Rate Forms. 
Engagement rates can be matched to activities by looking at the times 
when observations were made and the times when activities began and 
ended. For those activities which were not observed, an engagement 
rate must be estimated, either by taking the average (as in the example 
where, on 10-9, the engagement rate for spelling was estimated to be 
69%) or by taking the engagement rate of the most similar activity 
observed (as in the example, on 10-10, where the engagement rate for 
language was estimated to be the same as that of spelling, 80%). 
Student engaged time is calculated for each activity by multiplying the 
time in minutes by the engagement rate. These numbers are then added 
to find a total which is recorded on the front. In the example on 2T5^ 
on 10-9 the student engaged time is calculated as follows: 



106 minutes x .69 = 
45 minutes x . 70 = 
20 minutes x . 68 = 

TOTAL 



73 minutes 
32 minutes 
14 minutes 

119 minutes 
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The student engaged time for the third day of observations is 
calculated in a similar manner, using another small table on the back 
since two observations were made on that day. 

Finally, the teacher finds the average student engaged time thus 
far for the year by adding all the student engaged times (101 + 119 + 
79 = 299) and dividing by the number of days on which observations 
were made (299 v 3 - 100). 
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C. Selecting a data base (10 minutes) 



Rationale * Participants need to know what graphs and data bases are 
available to use In a comparison and what considerations to weigh In 
selecting graphs or data base for comparison* 



I4 Background considerations. (5 minutes) 

Data from Stalllngs and Kaskowltz study of Follow Through 
Eligible Classrooms Csee note C»3»d In Section 1] based 
on — 

(1) All students In 100 first grade classrooms and 58 third grade 
classrooms—variety of curriculum and classroom organizations 

(2) Children from low-Income families 

(3) Wide variety of geographic locations and variety In 
ethnic composition of classes 

(4) Posttest achievement measure was the Metropolitan Achieve- 
ment Test (MAT), a norm referenced^ standardized achieve- 
ment test 

b. Data from Beginning Teacher Evaluation Study — Phase III-B 
^'(1) Twenty-one fifth grade teachers who were volunteers 
^ (2) Urban and suburban classrooms In San Francisco Bay area 
^ (3) About 120 students (roughly 6 per classroom) were 



studied 

(a) 43% girls; 57% boys 

(b) 30% non-white; 70% white 

(c) Socioeconomic status — roughly! 

(1) 10% of parents were executives, managers, or 
professionals 

(11) 28% of parents were seml-professlonals » 
clerical^ sales, or technical workers 

(111) 47% of parents were skilled or seml-skllled 
workers 

(Iv) 15% of parents were unskilled workers 



Materials 



Strategy 



*2T10 — Some Selection 
Guidelines 



Review characteristics of sample 
classes from which data base was 
derived* Then review the selection 
guides using 2T10* 
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(d) Achievement— almost all students fell within the 
30th to 60th percentiles in achievement in reading 
and mathematics 

(4) Achievement measures 

*Ca) Criterion-referenced batteries for reading and mathematics 

* (b) Developed hy the Beginning Teacher Evaluation 
Study to reflect typical grade 5 content 

(c) Administered in October and the following May 

(d) Reading items only 
(i) Syllables 

(ii) Synonyms 
(iii) Word meaning in context 
(iv) Comprehension 

• Literal 

• Inferential 
•Main idea — literal 

• Main idea— inferential 
•Topic sentence 
•Fact and opinion 

(e) Mathematics items 

(i) Multiplication— products and missing factors^ 
some items timed 

(ii) Division 

(iii) Fractions 

(iv) Computation-including patterns and properties 

(v) Word problems 

(vi) Geometry 

^c* Degree to which your class and achievement measures are like 
those involved in study is an important consideration 

2» Some selection guidelines (5 minutes) 

^a. Subject considerations—separate graphs/data source for 
reading/language arts and mathematics 

^b. Grade level considerations 

(1) Graphs from Stallings and Kaskowitz include grades 1 and 
3 only; Beginning Teacher Evaluation Study data for 
grade 5 
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(2) Final selection based on teacher judgment regsrdlng 
similarities between his/her classroom and classrooms 
from research study 

(3) Suggested guidelines (2Tlp) 

(a) Kindergarten— first grade graphs 

(b) First grade~flrst grade graphs 

(c) Second grade — first or third grade graphs 

(d) Third grade— third grade graphs 

(e) Fourth grade — third grade graphs or fifth grade 
data from BTES 

(f) Fifth grade— fifth grade data from BTES 

(g) Sixth grade and above — BTES fifth grade data 

(^) No graph for fifth grade math since relationship of 
student engaged time to total math achievement not 
significant 
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SOME SELECTION GUIDELINES 

Consider Subject 
Math 

Reading/Language Arts 

Consider Grade Level— Final selection always based on 
teacher judgment regarding similarities between his/her 
classroom and classrooms from research study 

Kindergarten First Grade Graphs 

First Grade First Grade Graphs 

Second Grade First or Third Grade Graphs 

Third Grade Third Grade Graphs 

Fourth Grade Third Grade Graphs or 

BTES* Fifth Grade Data 

Fifth Grade BTES* Fifth Grade Oata 

Sixth Grade or Fifth Grade BTES* D^ta 
Above 

'^Beginning Teacher Evaluation Study 
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D. Comparing data to research (30-50 minutes) 



Rationale. Participants need to know how to read and interpret the 
graphs. They need to learn how to decide if they want to change 
student engaged time and how to set tentative goals for student 
engaged time. 



Materials 



Stra tegy 



*2TU~First Grade Reading 

and Word Analysis 

Total Graph 
:*'2T12~Third Grade Reading 

and Language Total 

Graph 

* 2T13~Reading a Graph 
2H26— Graphs 
(a-f) 

2Hl^ — Reading the Graphs 



Tfr2T15 — Considerations in 
Setting a 
Tentative Goal 



Explain the parts of each graphs 
using 2T11 as an example of a linear 
graph. Illustrate these points by 
usiug 2T12 as an example of an 
inverted U graph. 



Demonstrate how to find the expected 
achievement zone for any given level 
of student engaged time. Have 
participants identify the student 
engaged time zones corresponding to 
the expected level of achievement 
for each graph on 2H14» 

Discuss considerations in setting a 
tentative goal for student engaged 
time. 



2H6 --Summary Sheet for 
(a-b) Time 

2H16 — Comparing Your 

Data to Research 
(l)/2H17 — Comparing Data 

(a-b) to Research 
© 2T18— Key for /2H16 



Have teachers complete 2H16 for their 
own collected data; they should also 
indicate the zone corresponding to 
each day's student engaged time in 
the right margin of their own com- 
pleted Sununarv Sheet for Time. 
Leaders will complete the checkpoint 
instead. 



★l. Parts of the graph ( 2T1K 2l^26a ) (10-15 minutes) 
a. Identifying information 

(1) Grade shown in upper left corner 

(2) Title shown at top 

h. Relationship between student engaged time and achievement 
shown by straight line 

c. Student engaged time on bottom (horizontal axis)"amount of 
time in which students are actively engaged in a subject 
area 



Stiulrnt ;ich1c*vcmc*nt on s Lck* Cvc*rt Lcjil nx Is) — dl f f i^ronro from 
t*jtpo<?tt*il rnw scort' 

(1) Students take pretest"raw score Is number of right 
answers 

(2) Performance on later test (posttest) predicted on bnnls 
of relationship between pretest and poHttest scores 

(3) Difference between actual posttest score and predicted 
posttest score (actual * predicted) 

(a) Shown on side (vertical axis) 

(b) TeXXs whether above (positive) » at (zero), or 
beXow (negative) expected level of achievement 

2ones^-bars shown on each axis corresponding to above, at, 
or below expected raw score 

(1) If student engaged time falls In range marked by a bar 
at bottom, then chances are at least 2 out of 3 that 
student achievement will fall In area marked by same 
type of bar at side 

(2) No difference from expected raw score ( 2T1 I ) 

Solid black bars 

(1) Student engaged time of about 109-130 minutes 

(ll) Difference from expected raw score of about 
zero 

(b) If student engaged time falls in range marked by 
black bar at bottom, then chances are at least 2 
out of 3 that student achievement will be the same 
as expected as marked by black bar on side 

(3) Above expected raw score ( 2T11 ) 

(a) Shaded bars 

(1) Student engaged time of 130 minutes and above 
(li) Positive difference from expected raw score 

(b) If student engaged time falls In range marked by 
shaded border at bottom, then chances are at Jc^asL 
2 out of 3 that student achievement will be above 
expected level as marked by shaded bar on side 

(c) As student engaged time Increases within the zone, 
the likelihood that student achievement will be 
above the expected level increases 
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(4) Below expected raw s core (XLi-D 
(a) Slashed bflra 1N^^S^^^S^^ 



(1) Student engaged time of about 109 minutes and 
below 

(ll) Negative difference from expected raw score 

(b) If student engaged time la In range marked by 
slashed bar at bottom^ then chances are at least 
2 out of 3 that student achievement will be below 
expected level as marked by slashed bar on side 

(c) As student engaged time decreases within zone» 
likelihood that student achievement will be below 
expected level Increases 

tf* Minimum change units 

(1) Shown at bottom left of each graph 

(2) Smallest change In student engaged time that la likely 
to be reflected In student achievement 

(3) Example ( 2T11 ) 

(a) Minimum change — 10 minutes 

(b) Change less than 10 minutes may not really be a 
change because of measurement error 

g. Example of different graph shapes 

(1) So far have been looking at positive linear graph f ^Tl 1 ) 

(a) Increases In student engaged time are accompanied 
by Increases In achievement 

(b) Straight line shows more Is better 

(c) Same change In student engaged time always has same 
effect — Increase of 20 minutes In student engaged 
time (from 40 to 60 minutes or from 100 to 120 
minutes or from 180 to 200 minutes) accompanied by 
same change In difference from expected raw score 
(Increase of about 3 points) 

(2) Inverted U graph — Third Grade Reading and Language 
Total (2112) 

(a) Moderate amount of student engaged time corresponds 
to highest achievement 

(b) Same change In student engaged time has different 
effects when made at different points on graph 

(1) Increase In student engaged time from 50 to 
75 minutes Increases student achievement 
about seven points 

(ll) Increase In student eng^ed time from 125 to 

150 minutes does not change student achievement 
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(iii) Increase in student engaged time from 150 to 
175 minutes decreases student achievement by 
about 3 points 

(c) Five zones for student engaged time 




(i) Above expected raw score ( [r;^>::!<j'S:j''!---^'^ -i^----] ) — 
corresponding student engaged times of about 
113-172 minutes 

(ii) At expected level ( I^^^^H ) 

• Two zones 

•Corresponding student engaged times of 
about 88-113 minutes and 172-198 minutes 



(iii) Below expected raw score ( ) 



• Two zones on far left and far right 

•Corresponding student engaged times of 
below 88 minutes and above 198 minutes 

*2, Reading the graphs (5-10 minutes) 

a* Graph helps find expected achievement for a classroom given 
the level of student engaged time 

b. Example (2T13) 

(1) Locate student engaged time (80 minutes) on bottom 
(horizontal axis) 

(2) In slashed zone so below expected raw score 

(3) Draw vertical line up to straight line 

(A) Draw horizontal line across from point on line to side 
(vertical axis) 

(5) Difference from expected raw score is where line inter- 
sects vertical axis 

(a) About halfway between 0 and -9 

(b) Closer to -9 so probably about -5 units 

c, Practj.ce ( 2H14) 

3- Fifth grade math—no graph ( 2H26f ) (2 minutes) 

a* Relationship between student engaged time and total math 
achievement not significant 

bt Relationship between student engaged time and achievement 
in fractions^ multiplication, geometry, and decimals was 
significant — more is better 

A, Setting a tentative goal for student engaged time (5-10 minutes) 

a« Some suggested guidelines 

(1) Probably want to change by at least minimum change unit 
(shown below each graph) if any change is made 
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(2) Changes of at least the miniiDum change unit made across 
zones are likely to affect student achievement 

(3) Changes within zones may or may not bring about changes 
in student achievement 

b» Considerations in setting a tentative goal (2T15) 

(1) Similarity of students to those in research study 

(a) First and third grades — Follow Through eligible 
classrooms 

(b) Fifth grade — students at 30th to 60th percentiles 
in reading and math achievement 

(2) Desired level of achievement — probably want student 
engaged time to be at least within zone corresponding 
to expected level of achievement 

(3) Importance of student engaged time — research has shown 
to be significantly related to student achievement 

(4) Relative importance of subject area to district^ ftchool^ 
teacher^ students* needs — amount of time available 

★ 5. Own data f 2H16 ) or example f /2H17 . 2T18) (10-15 minutes) 

.a» Graphs — titles of selected graphs 

b» Student engaged times — from Summary Sheet for Time 

c» Zone — check column indicating in which area student engaged 
time falls and indicate by 0 or - on Summary Sheet for 
each day 

d» Tentative goal 
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MATERIALS 



In addition to the materials included here, the following will 
he. needed: 

2H26— Graphs 
Ca-f) 

2H6 — Summary Sheet for Time 
(a-b) 
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READING THE GRAPHS 



Consider three third grade classes 4 The student engaged time In math for 
each class is shown below. Use the Third Grade Math Total Graph to locate 
student engaged time on the bottom axis and check the column corresponding 
to its zone* 



Class 


Student Engaged Time 


' ■ - - 

Zone 


Above 


At 1 


Below 








A 


45 iiiln* 








B 


' — — — — — 

72 min* 








C 


96 min* 










Esther Freedman is a third grade teacher at the 
fictional New Delpen Elementary School* Find her 
average student engaged times on the bottom axis 
of each graph and check the column corresponding 
to the zone* 







Zone 


Graph 


Student Engaged Time 


Above 


At 


Below 










Third Grade 
Reading and 
Language 
Total 


96 min* 


■ 






Third Grade 
!lath Total 


39 min. 








■ ■ ■ ■ ■ ■ ■ ■ 1 ■ -fc-- - — ~ — — - — - -p - — - — — ■ 









&aoqy 



(3) ^ao<iV 



(3) 



(V) 
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CONSIDERATIONS IN SEHING A TENTATIVE GOAL 



Sone suggested guidelines: 



* Most teachers will probably want to change by at 

least the nininuin change unit if any change is 
made. 

* Changes of at least the nininum change unit made 

across zones are likely to affect student 
achievement 

* Changes within zones nay or nay not bring about 

changes in student achievement 




* importance of student engaged time 



• Relative importance of subject area--antounl of lime 

available 

• Desired level of achievement 

• Similarity of your students to those in research 

study 



2.55 
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COMPARING YOUR DATA TO RESEARCH 



List the title(s) of the graph(s) you have selected for use in comparison in the first column. Record 
your most recent average student engaged time from your Summary Sheet for Time in the second column. 
For each of the selected graphs, indicate the zone in which student engaged time falls by checking the 
appropriate column. Set a tentative goal for student engaged time for reading/ language arts and for 
math by considering the graphs, the zone in which achievement falls, and the minimum change unit shown 
below each graph. 



— 

Title of 
Selected Graph 

1 — , 


Student 
Engaged 
Time 


Above 


Zone 


Igelow 

M 


Tentative Goal 
for Student 
Engaged Time 




































. , ^ ^ 
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COHPARINC: DATA TO RKSKARCH 



Tony Casinl, a third grade teacher at the fictional New Delpen Fllementary School, 
has collected information in his classroom and completed his calculations. The 
front of his completed Summary Sheet for Time is shown on the next page* 



The titles of the graphs Tony has selected are shovm in the first column below. Record his most recent 
average student engaged time in the second column* For each of the graphs, indicate the zone in which 
student engaged time falls by checking the appropriate column. Set tentative goals for Tony for student 
engaged time for reading/language arts and for math by considering the graphs^ the zone in which achieve- 
ment falls, and the minimum change unit shown below each graph. 




Title of 
Selected Graph 


Average 
Student 
Engaged 
Time 


Zone 


Tentative Goal 


Above 


: At 


fielow 


for Student 






m 


Engaged Time 


Third Grade Reading and 
Language Total 












Third Grade Math Total 










^ , „ , — 
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Teacher CflLS»rn* 



SUMMARY SHEET FOR TIME 




Scace <^_ 03 School J Q9 Tcnchlng Day 300 



Dlacrlcc n Teacher ff SSHft . Year l'^iq'\'tftO 



RBADINC/LAMCUACE ARTS 



Dace 


Coder ilF 


pArc o£ 
Period 


LngagemenC 

Race 


AiiocAcea 
Tine 


Scudcnc 
Engaged 
Tloe 


Average 
Scudeac 
Engaged Time 










ISO 
















\'bO 






etas 




15% 






I30 


































































































- 




MATH 



DaCe 


Coder ^ 


Pate of 
Period 


Eagagemeat 
Race 


AllocaCed 
Time 


Scudenc 
Eagaged 
Time 


Ave rage 
Scudenc 
Engaged Time 












3C 








































































































1 
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KEY FOR /2H17 



Title of 
Selected Graph 


Student 
Engaged 
Time 


Zone 


Tentative Goal 
for Student 
Engaged Time 


+ 


0 




Third Grade 
Reading and 
Language Total 


130 min. 




X 




AT LEAST 
130 min. 
(No change) 


Third Grade 
Math Total 


33 min. 






X 


AT LEAST 
41 min. 



87 

2.59 



2.60 



9/3/ftn 



D.l.d. Student Achievement — Difference from Expected Raw Score 

This note provides a technical explanation of the differences from 
the expected raw score. 

Suppose the third grade students in a school were grouped into tliree 
classes according to their previous year's reading achievement test scores 
high, medium, and low reading level classes. Suppose, also, that they re-* 
ceive similar instruction for the same amount of time using appropriate 
instructional materials. If a posttest were administered to them at the 
end of the school year, one may easily predict that the posttest mean of 
the high reading level class would be greater than that of the medium 
reading level class^ which^ in turn ^ would be greater than that of 
the low reading level class. Even if special reading instructions were 
given to certain classes, the posttest mean scores would still depend 
largely upon their pretest mean scores (initial reading level). 

The difference from the expected raw score (residual score) is the 
remaining posttest score after the difference on the pretest (initial 
reading levels) has been statistically controlled. That is, given pretest 
and posttest scores for numerous classes at a given grade level, a best-- 
fitting graph of the relationship between them can be drawn. 



Such a graph provides a way of locating the predicted posttest score for 
a given pretast score. By drawing perpendicular lines from the bottom 
axis to the graph of the relationship and then over to the vertical axis, 
the predicted posttest score can be found for a given pretest score. 
Such a graph may look like the one above. For a pretest score of 30, the 
predicted posttest score is approximately 35. Wlien the predicted poctte^^t 
score is subtracted from the actual posttest score for that classroom, 
the difference from the expected raw score is called the residual score. 
Now, it can be seen that a 0.0 difference from the expected raw score 
does not necessarily mean that there was no gain but, rather, that the 
gain was precisely what was predicted. It is also clear that a negative 
difference, as -9.0, does not necessarily mean that there was actually a 
decrease in achievement. It does mean that the actual gain (if any) was 
less than the predicted gain. Or, to put it another way, the students 
did not learn as much as was predicted for students with that pretest 
score. 



Posttest 
Score 




10 • 



10 20 30 40 50 



Pretest Score 



69 
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The specific relationship of the residual scores in the Stallings 
and Kaskowits study to percentile scores, grade equivalent scores, or 
£;tanxne scores Is not known. 

To obtain residual scores, a predicted posttest score is calculated 
for each pretest score from the linear regression of the posttest scores 
on the pretest score:: . Then a residual score is calculated by subtracting 
the predicted posttest score from the actual posttest score. For each 
classroom, then. 

Residual Score = (Actual Posttest Score) - (Predicted Posttest Score) 

The predicted posttest score for each classroom is obtained by means of 
the following formula. 



Technically, the residuals are defined as the n differences 
d^ - Y^' , i - 1, 2, . . . , n, where Y^ Is an observation and Y^ ' is 

the corresponding fitted value obtained by use of the fitted regression 



The residuals are the differences between what is actually observed 
and what is predicted by the regression equation — that is, the amount the 
regression equation has not been able to explain. For more information, 
see Applied Regression Analysis by N. R, Draper and H. Smith (New York! 
Wiley, 1966), p. 86. 




Prete 
Mean 
Scor 



Scores 



equation. 
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D. I.e. Zones 



On each graph, student engaged time (horizontal axis) is marked with 
three kinds of zones (bars) which correspond to the positive, zero, and 
negative zones on the student achievement scale (vertical axis showing 
difference from the expected raw score). Generally, if the student 
engaged time of a classroom falls within the range marked by a bar on the 
horizontal axis, then it is likely (at least two out of three times) that 
student achievement will fall in the corresponding range on the vertical 
axis. 

The zones were derived from the shape of each graph and a statistical 
measure.* When a graph developed on the basis of classroom data is used 
for making a prediction (e.g., predicting student achievement from student 
engaged time), some errors are always expected (unless the relationship 
is perfect). When these errors of prediction are taken into consideration, 
the deviations of the values of student achievement predicted from all of 
the points in the "zero" student engaged time zone from a zero difference 
from the expected raw score are all quite small. These deviations can 
essentially be regarded as zero on the student achievement axis. The 
positive and negative zones were placed around the "zero" student engaged 
time zone(s) according to the shape of the graph. 




*An estimated standard error of prediction was the basic statistic used 
to determine the zones. For the procedures used to calculate the zones^ 
see Rim, E., 6t Segars, J. A conceptual framework for the evaluation of 
the RBS basic skills instructional improvement effort in the tri-state 
region . Philadelphia, Pa.: Research for Better Schools, Inc., 1979. 
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D*l.f. Minimum Change Unit 




A rainimura change unit is specified at the bottom left of each graph. 
This miTilmum change unit is the smallest amount of change in student 
engaged time that is likely to produce a change in student achievement. 
The minimum change unit is based on a statistic* that indicates the 
uncertainty involved in predicting student achievement from student 
engaged time* The value of this statistic varies from graph to graph 
because the uncertainty in prediction varies from graph to graph and test 
to test* For each graph, the minimum change unit has been established so 
that if student engaged time changes by that amount, the chances are at 
least two out of three that there will be a change in student achievement* 
If changes in student engaged time are less than this unit, changes in 
student achievement are much less likely to occur* 




*An estimated standard error of prediction was the basis of computing 
each minimum change unit* For the computational procedures involved, 
see Rim, E* , & Segars, J* A conceptual framework for the evaluation 
of the RBS basic skills instructional improvement effort in the 
tri-state region . Philadelphia, Pa*: Research for Better Schools, 
Tnc*, 1979. 
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E, Making a decision (30-50 minutes) 



Rationale . Participants need to examine allocated time and engagement 
rate in order to determine tentative goals needed to reach their goal 
level of student engaged time. They need to consider the reasonable- 
ness and value of such changes, revising their tentative goal for 
student engaged time If necessary. 



Materials 



St rategy 



*2T19— Making a Decision 
2T20 — Different Ways of 
Getting 90 Minutes 
of Student Engaged 
Time 



Provide an overview of the activities 
involved In making decisions about 
student engaged time* 



— Rulers 
*2T21 — Setting Engagement 
Rate Coals 
2H22— Tony Caslni's Goals 
(a-b) 

2H16 — Comparing Your 

Data to Research 
2H6 — Summary Sheet for 
(a-b) Time 
★ 2H23"Maklng a Decision 
(a-b) 

(l) 2H2A~Making a Decision 
Ca-c) 

© 2T25— Key for 2H2A 
*2HA — Time Nomograph 



Review the research on engagement rate* 
Discuss the example on 2H22* Have 
teachers complete Table 1 on 2H23 for 
their own classrooms while leaders 
complete Table 1 on 2H2A* 



Explain how to use the nomograph (2HA) 
to find the tentative goal for allo- 
cated time corresponding to the tenta- 
tive goals for student engaged time 
and engagement rate* Look at the 
example on 2H22 and then have teachers 
complete Table 2 on 2H23 for their own 
classrooms while leaders complete 
Table 2 on 2H2A* 



Discuss considerations Involved In 
deciding whether the tentative goal 
for allocated time is feasible* Look 
at the example on 2H22 and have par- 
ticipants decide whether allocated 
time goals are feasible for their own 
classrooms (teachers use 2H23) or for 
the checkpoint (leaders use 2H2A), 
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Overview (?Tl . 9 ) (5 minutes) 
a» Current status 

(1) Have tentative goal for student engaged time 

(2) Student engaged time = allocated time x engagement rate 

(a) Given level of student engaged time (90 minutes) 
may be result of different levels of allocated 
time and different engagement rates (2T20) 

(i) Allocated time of 90 minutes and 100% engage^ 
ment rate—engagement rate not likely 

(ii) Allocated time of 120 minutes and 75% engage- 
ment rate^'-'allocated time could he ^juced or 
student engaged time increased acn. ^y 
increasing engagement rate 

(iii) Allocated time of 180 minutes and 50% engage^ 
ment rate — allocated time could be reduced or 
student engaged time increased considerably 
by increasing engagement rate 

(iv) Allocated time of 300 minutes and 30% engage- 
ment rate — allocated time not likely 

(b) Need to look at both allocated time and engagement 
rate in order to decide on student engaged time 
goals 

b. Step 1 

(1) Look at current engaganent rate 

(2) Use teacher judgment and research findings to set 
tentative goals 

c. Step 2 

(1) Given tentative goal for student engaged time and for 
engagement rate, find allocated time goal 

f^^ I. J ^ - Student engaged tima 

(a) Allocated time = - JLr ' \ ' J r J^ ^ 

engagement rate 

(b; Use nomograph or calculator 

(2) Decide if tentative goal for allocated time is feasible 
for your classroom 

d» Step 3 — record or revise goals 

Setting tentative goals for engagement rate — step I f 2T21 ) 
(5-10 minutes) 

' a» Research findings 

(1) Average: 60% 
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(2) Range; 30-90% 

(3) Students with higher engagement rates usually learn tnorc 
^tb» Guidelines for setting an engagement rate goal 

(1) Probably want to set a tentative goal of at least an 
average engagement rate — 60% 

(2) Probably do not want to set a tentative goal above 
highest engagement rates found — 90% 

(3) In 60-85% range, probably want increase of at least 
5% if possible 

(A) Feasible for situation 

c» Examples 

^(1) Current average engagement rate is 50% — probably want to 
try to increase engagement rate to at least 60% 

(2) Tony Casini ( 2H22 ) 

(a) Current engagement rates — 80%, 75%, 70% 

(b) Tentative goal for engagement rate — Tony sets at 
80% 

^ d» Setting tentative goal for engagement rate for own data ( 2H23 ) 
or example ( 2H2A . 2T25a ) by completing Table i 

(1) Daily engagement rates from Summary Sheet for Time 

(2) Set tentative goals for engagonent rates that are 
feasible for situation 

3» Looking at allocated time — step 2 (15-20 minutes) 

^a» Setting tentative goals for allocated time 

(1) Use nomograph or calculator to find (2H4, , 2H22b ) 

(a) Given tentative goal for student engaged time 
(60 minutes) 

(b) Civen tentative goal for engagement rate, or current 
engagement rate if no change is desired (80%) 

(c) Nomograph 

(i) Place straightedge on point for tentative goal 
student engaged time and for tentative goal 
engagement rate 

(ii) Point where straightedge crosses allocated 
time scale is tentative goal for allocated 
time (about 75 minutes) 

(d) If using calculator, divide student engaged time 
goal by engagement rate goal 
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★ (2) 



(3) 



(2) Finding tentative goal for allocated time for own data 

( 2H23 ) or example ( 2H2 A , 21; 2 , 5 ^ by completing Table 2 

(a) Tentative goals for engagement rate from last 
column of Table 1 

(b) Tentative goals * for student engaged time from 2H16 

(c) Tentative goals for allocated time computed by 
using nomograph (2H4) or calculator 

Decide if tentative goals for allocated time are feasible 

^(1) Change may affect other subject areas (e*gp, science, 
music, art, social studies) 

Change may conflict with school schedules or policies 

(a) Schedule for each subject may be fixed 

(b) Schedule for pullouts^ mugic, gym, etc*, may be 
difficult to change 

Example of Tony Casini (2H22) 

(a) Currently spends at most A5 minutes on math each 
day 

(b) Decides 75 minutes is not feasible because of lunch 
and music/gym schedules 

Deciding if allocated time goals are feasible for own 
data f 2H23 ) or example ( 2H2A , 2T25 ) 

Recording and/or revising goals — step 3 (5-15 minutes) 

a. If allocated time goal is feasible, record goals 

b* If allocated time goal is not feasible, revise f^oals 

(1) Decide if allocated time goal can or should be revised 
and record 

(2) Check to see if engagement rate goal can be changed — 
if so, revise it and record 

(3) Calculate student engaged time goal (allocated time x 
engagement rate), compare to original goal, and record 

c. Example of Tony Casini — revises goals f2H22c ) 

(1) Sets allocated time goal of 65 minutes 

(a) Current allocated time — A2 minutes 

(b) Goal found in step 3 — 75 minutes 

(c) Revised goal closer to original goal 



★ (A) 
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(2) Increases engagement rate goal to 85X 

(a) Current engagement rate — 75% 

(b) First goal— 80% 

(3) Calculates student engaged time goal — about 55 minutes — 
close to original goal of 60 minutes 



d. Recording and/or revising goals for own data' (2H23) or example 
(2H24, 2T25) . 
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MATERIALS 



In addition to the materials Included here^ the following will 
be needed: 



2H16 — Comparing Your Data to Research 

2H6 — Summary Sheet for Time 
Ca-b) 
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MAKING A DECISION 



STEP 1 



Look at current engagement 
rate and set a tentative 
goal that is feasible for 
your classroom 




STEP 2 



STEP 3 



1 



Find tentative goal for 
allocated time by using 
nomograph or calculator 
and decide if it is 
feasible for your class- 
room 



I 



Record or revise goals 



• 

1^ 
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SETTING ENGAGEMENT RATE GOALS 

♦ Research Findings 

Average: 60% 
Range: 30-90^ 

Students with higher engagement rates usually 
learn more. 

9 Guidelines for setting an engagement rate goal: 

Most teachers will probably want to set a 
tentative goal of at least an average 
engagement rate (60/o). 

Most teachers probably will not want to set 
? tentative goal above the highest engagement 
rates found (90%). 

In 60-85iv range, most teachers will probably 
want to increase engagement rates by at least 
5% if possible. 

Most teachers will probably want to use their 
own Judgment in considering the feasibility 
of tentative goals for their own particular 
situations. 



SI 
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TONY CASTNI'S COALS 



Tony Casini, a third grade teacher at the fictional 
New Delpen Elementary School, has collected 
information in his classroom about student engaged 
time and has completed his calculations* He has 
set a tentative goal for student engaged time in 
math of 60 minutes* 





Step l! 

Setting Feasible Goals for Engagement Rate 



The engagement rates for Tony's classroom are 80%» 75%, and 70%* 
Tony decides that he wants to increase his engagement rate to 80%* 



Step 2: 

Computing Tentative Goals for Allocated Time 




Tony records this tentative goal for engagement rate and the tentative 
goal for student engaged time which he found earlier in the table 
below* 

Find the corresponding tentative goal for allocated time, using the 
nomograph (2H4) or a calculator . ^ -r * student engaged time x 

(<^^^ocated ttme = -^^-^igmS^t rate ^' 



Tentative Goal for 
Engagement Rate 


Tentative Goal for 
Allocated Time 


Tentative Goal for 
Student Engaged Time 


■ ■ ' ■ 

80% 


^^^1- ■ II ■ 11 


60 minutes 



Deciding if Allocated Time ^oals are Feasible 

In order to decide whether this allocated time is feasible, Tony first 
examiiies his current situation* Tony*s allocated times for math on 
the days on which information was collected were 45* 42, and 40 
minutes* Tony decides that the goal of 75 minutes is not feasible for 
his classroom because of lunch and music/gym schedules* 




's&q.nu'iiu q.noqv s^ qvoB aiuzq. pa^vvo^^V — ? d^<l9 
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Step 3: 

Recording and/or Revising ^^"^.^^.^ 



Since Tony decided that the goaX for allocaCtid time of 75 minutes was 
not feasible, he must revise his goals* 



• Tony first decides that he can Increase allocated time to 
65 minutes. He records this revised goal In the table 
below. 

©Next Tony decides that he can probably increase rils engage- 
ment ratB to 85J;. He records this revised goal In the 
table below. 

% Find Tony's new goal for student engaged time by using the 
nomograph (2H4} or a calculator (allocated time x engage- 
ment rate)* Record this goal below* 



Engagement Rate 


Allocated Time 


. Student Engaged Time 


85% 


65 minutes 





Tony finds that this revised goal Is slightly less than his original 
goal of 60 minutes* He decides to try to reach this revised goal for 
student engaged time In math and completes a similar sequence of steps 
to find goals for reading/language arts. 



*s&:inu'iiu gg sz ^voB 0unii p&6vdu& :iu&pn:is p^s-iaBu — £ d0q.g 
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MAKING A DECISION 

Step 1; Setting Eng<igement Rfite Goalfl 

• Record your rurn^nt oii^:^;i^t^mcnt rvitc^; { from your Summer v 
Sheet for Time) in the first column. 



• Set tentative engagement rate goals that are feasible 

for your classroom. You will probably want to set goals in 
the range of 60-90%, with an increase of at least 5% if 
possible. 





Step 2 



Talkie i 


Daily 
Engagement Rate 


Tentative Goal for 
Engagement Rate 


Reading/Language Arts 




% 


% 


% 


- . Math 


% 


% 


% 


% 


2: Finding Allocated Time Goals 

•Record below your tentative goals for engagement rate 



(from Table O . 

• Record your tentative goals for student engaged time 
(from 2H16). 

• Find the corresponding tentative goals for allocated time 

by using the nomograph (2H4) or a calculator 

^ , . student eneaeed time. 

(allocated time = -) 

engagement rate 



Table 2 


Tentative Goal for 
Engagement Rate 


Tentative Goal for 
Allocated Time 


Tentative Goal for 
Student Engaged Time 


Reading/ 
Language 

Arts 


% 






— > 

Math 


. — . . . * — — _ — 

% 


^ 





• Decide whether these allocated time goals are Tensiblc. 




1^ 



2.7G 5^ 



Step 3: 




Recording and/or Revising Goals 

♦ If so, go on to tlu^ next phase. Selection nnd Pr<*pnrn t ion , 

♦ If an allocated time is not feasible: 

First consider whether your current allocated time can ^^^^^^^^ 
or should be changed so as to be closer to the tentative 
goal established in Table 2, Record the allocated time goal 
below. 

Ne?ct reconsider your engagement rate goal in the last column 
of Table 1* If this goal can be revised, do so» Record 
your goal for engagement rate below. 

Find the corresponding student engaged time goals by using 
the nomograph (2H4) or a calculator (allocated time x 
engagement rate). Compare this to your original goal^ 
going back to the graph if there is a difference of more 
than 5 minute^ and record it below. 



Table 3 


Allocated 
Time 


Engagement 
Rate 


Student Engaged 
Time 


Read i n g /Lang ua ge 
Arts 




% 




Mathematics 




% 






So 
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KAKING A DECISION 

Jim Hetndricks teaches a below-average fourth grade 
class at New Delpen Elementary School in Eastern School 
District, Atlantic State. The principal and teachers 
in the school want to concentrate primarily on reading/ 
language arts and math this year. The front of his 
completed Summary Sheet for Time is shown below. 



Stat* A+Ur^-^'c ! 
District £<*6+ie,rr\ * 




SUMMARY SHEET FOR TIME 






School l^tu^X>&\^rk! State 9 0 2> 
Teacher HcmiinLkil District # Ml 


School # 0% 






Teacher ^ \0\t> Year l^31^l^«2- 
















READI^G/LANGIJAGE ARTS 












Dat^ 


Coder # 


Pace oE 
Period 


Engagenenc 
Rac^ 


Allocated 
Tltne 


Student ' 
Engaged 
Time 


Avi3 ra^i3 
J Cuderc 


















^^^^ 












10% 




























MATH 
















Dace 


Coder J 


Pare of 
.Period 


Engagement 
Rata 


Allocated 
Time 


Student 
Eagaged 
Time 


Ave rage 
Engaged Tir-i 










m- 




- 






















33 min* 





































.Jim iist?<J thti third-grade graphs to set tentative gonls for student 
(?ng<i^t^d time in read 1 nj;/ language arts and matlu 



Title of 


Student 


Zone 


Tentative Coal 


Solertcd nrapii 


Kngngc^d 


Above j 




Below 


for Student 




Time 






////^ 


Engaged Time 




Third Crade 
Reading and 
Language Total 


110 mln. 




/ 




130 mln. 


Third Crade 
Math Total 


36 min. 








50 min. 



be ch.ingod somewhat. 
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Step 1: Setting Engagement Rate Goals 

• Record Jim's current engagement rates (from his Summary 
Sheet for Time) in the first column. 

• Set tentative engagement rate goals which might be feasible 
fcr his classroom, you will probably want to set goals in 
the range of 60-90%, with an increase of at least 5% if 
possiole . 




T^tbLe 1 


Daily 
Engagement Rate 


Tentative Goal for 
Engagement Rate 




% 




Reading /Language Arts 


% 


% 




t<f 




% 




Math 


% 


% 




% . 






Step 2: Finding Allocated Time Goals 

• Record below Jim's tentative goals for engagement rate 
(from Table 1) . 

• Record his tentative goal for student engaged time (from 
the table on 2H24a) . 

• Find the corresponding tentative goals for allocated time 

by using the nomograph (2HA) or a calculatf>r 

/ ^ , f student ongagi*d t. (mo. 

(allocated time = — ). 

engagement rate 



Table 2 


Tentative Goal for 
Engagement Rate 


Tentative Goal for 
Allocated Time 


Tentative Goal for 
Student Engaged Time 


Reading/Language Arts 


% 


- 




Math 


% 







• Decide whether these allocated time goals are feasible. 
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Step 3: Recording and/or Revising Ooals 



If Jim's allocated time goal in a subject area is feasible, 
go on to the next phase, Si^lectlon and Preparation* 

If his allocated time goal in a subject area is not feasible: 

• First consider whether Jim's current level of allocated 
time can or should be changed so as to be closer to the 
tentative goal established in Table 2. Record the 
revised allocated time goal in Table 3» 

• Next reconsider Jim's engagement rate goal in the last 
column of Table 1. If this goal can be revised, do so. 
Record his goal for engagement rate (revised or not) 
be low » 

• Find the corresponding student engaged time goals by using 
the nomograph (2H4) or a calculator (allocated time x 
engagement rate) » Compar*i this to Jim's original goal in 
the last column of Table 2. If there is a difference 
between the tv/o goals of more than 5 minutes ^ go back to 
the graph » 



Table 3 


Allocated 
Time 


Engagement 
Rate 


Student Engaged 
Time 


Reading/Language Arts 




% 




Mathematics 




% 
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TABLE 1 


Daily 


Tentative Goal 
for 

Engagement Rate 


Engagenent Rate 


Reading/ 
Language Arts 


60% 




70% 


70-90% 




6S% 






55^ 




Hath 




70-90% 




70% 





2.31 
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KEY FOR 2H24 



TABLE 2* 


Tentative Goal 
tor 

Engagement Rate 


Tentative Goal 
tor 

Allocated Tine 


Tentative Goal 
for Student 
Engaged Time 


Reading/ 
Language Arts 




186 min. 


130 fflin. 




163 min. 




144 min. 


Math 


M 


71 min. 


50 min. 


80% 


63 min. 


m 


56 min. 



^Correct answers for three possible tentative goals for allocated 
time (corresponding to three different tentative goals for 
engagement rate) are shown. If you picked a different engagement 
rate goal, your answer for allocated time should be between those 
given for the goals above and below your own. Decisions as to the 
feasibility of these goals may differ. 
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E,3.b. Guidelines tor Setting an Engagement Rate Goal 

There nre two mnjor research studies which have examined engagemc*nt 
rntes, the percent of the class engaged in academic tasks* One of these 
studies, the Instructional Dimensions Study, found an average engagement 
rate of about 60% in first and third grade classrooms. This same study 
found that about one-third of the classes in the study had engagement 
rates above 70%» The other study. Phase III-B of the Beginning Teacher 
Evaluation Study, used a slightly different definition of engagement rate 
which disregarded the time spent in management and transition activities » 
The average engagement rates in this study of second and fifth grade 
classes were thus slightly higher, about 70-75%. About one-third of the 
teachers had engagement rates above 80%» The tange of engagement rates 
found in these studies is from about 30% to 90%. 

Since research indicates that students with higher engagement rates 
usually learn more, teachers will probably want to set tentative goals 
for engagement rate that are at least average (the average of 60% from 
the Instructional Dimensions Study is suggested since the definition 
used in this study more closely matches that used in information collection). 
Since no research study has found any classroom that consistently has an 
engagement rate above 90% and since some classroom time is needed for 
giving nonacademic directions and changing activities, teachers will 
probably not want to set goals above 90%» 

In order to help teachers ascertain how much change is meaningful, a 
standard error of estimate was calculated for each research study, using 
the following formula; 



These standard errors of estimate were approximately equal to 5% (4»9% - 
6.6%), It is thus recommended that changes of least 5% be made* 



Standard Error 
of Estimate 




Standard Deviation 
for Engagement Rate 




Reliability of 
Engagement Rate 



2. OA 
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The following graphs are included here; 

Reading/Language Arts 

Grade 1 — Reading and Vford Analvsis Total 
Grade 3 — Reading and Language Total 
Grade 5 — Reading Total 

Mathematics 

Grade 1 — Math Total 
Grade 3—Math Total 
Grade 5 — Math Total 



2.35 
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READING AND WORD ANALYSIS TOTAL 



GRAuE 



36 
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Student Engaged Time minutes) 

^[ ^n^mum change of at: least IQ minutes . — Above expected level 

of achievemen t 
— At expected achieve- 
Source; Stallings & Kaskowitz (1974) ment level 

— Below expected level 
of achievement 



2.86 
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READING AND LANGUAGE TOTAL 



18 
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Student Engaged Time ^^^te«) 

WM — At expected 

Minimum change of at least 13 minutes. — Above expected level of 



Source! Stallings & Kaskowitz (197A) 



level of 
achievement 



achievement 
— Below expected 
level '{>f 
achievement 




ERIC 



2.87 



95 



9/2/8C 



READING TOTAL 



GRADE 

5 



CD 
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Minimum change of at lease 7 minutes . 
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Source: Beginning Teacher EvaluaCion Study, 1978 



;:;] — Above expected level 
of achievement 
— At expected achieve- 
ment level 
— Below expucted achieve- 
tnenc level 
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mn TOTAL 



GRAuL 
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Minimum change of at least 6 minutes . 
Source: Stallings h Kaskowitz (1974) 



— Above expected level 

of achievement 
— At expected achievement 
level 

— Below expected level 
of a ch i evemen t 



2.89 
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MATH TOTAL 
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Student Engaged Tirrie minutes) 



Minimum change of at least 8 minutes . 
Source: Stallings & Kaskowitz, (1974) 



SS] — Above expected level 
of achievement 
— At expected achieve- 
ment level 
— Below expected level 
of achievement 
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NiATH TOT/l 

GRADE 



No significant relationship was found 
between student engaged time and 
students* total math achievement, 
with student engaged times ranging 
from 17 to 46 minutes. 

Significant relationships were found 
between student engaged time and 
achievement in certain subtest areas. 
The more engaged time, the higher 
achievement in fractions, multipli- 
cation, geonietry, and division. 



Source: Beginning Teacher Evaluation Study 
Phase 111-3, 1978 
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